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Executive Summary 

As the final stage of West Virginia’s rollout of the Next Generation Content Standards 
and Objectives (NxGen CSOs), the regional education service agencies (RESAs) conducted six 
train-the-trainer events in the spring of 2013 to prepare educators— mostly teachers— to pro- 
vide professional development back in their home schools and districts. These events, called 
Educator Enhancement Academies (EEAs), lasted two or three days depending on which 
RESA conducted them, and targeted teachers in grade levels that had not yet received profes- 
sional development in the NxGen CSOs, that is, Grades 2-3, 6-8, and 10-12. 

The first phase of this study looked at how well prepared those trainers were at the end 
of their EEA experience by asking them about their experiences during the training and after 
they, themselves, conducted training sessions during the summer of 2013. This study further 
examines the experience of those same participants in providing their own training, as well as 
what additional professional development they received from the RESAs. It also asks about 
the experience of the end-user teachers who received professional development from the EEA- 
trained teachers and other sources in the targeted grade levels during the 2013-2014 school 
year. 

Research Questions 

EQi. To what extent did participants in the EEAs follow up with their own training? 

EQ2. What challenges did EEA participants encounter and what supports did they indicate 
they needed going forward? 

EQ3. What follow-up did RESAs provide after the initial EEA training? 

EQ4. What was the NxGen PD experience of end-user teachers and EEA teacher trainers 
during the course of the 2013-2014 school year? 

EQ5. What were the outcomes of 2013-2014 professional development experiences in terms 
of teachers’ overall sense of preparedness to teach to the NxGen CSOs and perceived 
impacts of those experiences on knowledge, practice, and beliefs? 

EQ6. What training-related factors may have been at work to produce these outcomes? 

Methods 

We examined three populations in this phase of the Educator Enhancement Acade- 
mies (EEA) study: (a) 953 teachers, district office staff, and others who received training at 
the RESA-sponsored EEAs to become teacher trainers; (b) professional development directors 
or executive directors from the eight RESAs; and (c) general and special education teachers 
involved in teaching English/language arts (ELA) and mathematics across the state in Grades 
2-3, 6-8, and 10-12. For all groups, we surveyed the full population. 

Using three instruments, we collected data as follows: 

• For EEA trainees, we used the Follow-up EEA Participant Survey in September 2013; 
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• For RESA professional development directors, the RESA PD Director Interview Pro- 
tocol (May 2014); and 

• For ELA and math teachers in targeted grades across the state, we used the NxGen 

Standards Professional Learning Survey (April-May 2014). 

Results 

Of the 953 EEA participants, 599 responded to the Follow-up EEA Participant Survey, 
for a 63% response rate. Of the 4,686 ELA and math teachers in the targeted grades, partici- 
pants returned 1,662 usable responses to the NxGen Standards Professional Learning Survey, 
for a 25% response rate. Although this is a lower response rate than we typically see-probably 
due to testing and other pressures on teachers during the April-May survey period-the total 
number of responses fell only slightly short of our calculated target sample size (1,740). PD 
directors or executive directors responded to the interview protocol for all eight RESAs. Find- 
ings are summarized and interpreted below by evaluation question. 

EQ1. To what extent did participants in the EEAs follow up with their own training? 

The great majority of the respondents (85%) to the September 2013 follow-up survey 
indicated they had provided training. Depending upon EEA participants’ regular role in their 
counties, they were able to provide more or less training to end-user teachers. District central 
office staff seemed in the best position to provide training as evidenced by the numbers they 
trained, the number of hours they had provided to a typical participant in their sessions by 
early fall 2013, and the additional hours of training they planned to provide. By comparison, 
other role groups appeared at a disadvantage, especially teachers and principals with regard 
to planned follow-up. 

EQ2. What challenges did EEA participants encounter and what supports did they indicate 
they needed going forward? 

Most EEA-trained trainers were able to schedule at least some training; where training 
had taken place, scheduling was still difficult, as was getting people’s attention and buy-in. 
Many of the trainers reported needing more training themselves, especially in the instruc- 
tional shifts and other aspects of classroom implementation, and help in locating curriculum 
and other resources they needed for planning instruction. 

EQ3. What follow-up did RESAs provide after the initial EEA training? 

EEA-facilitated follow-up training took place in schools, counties, and RESAs, alt- 
hough counties seemed to be squarely in the lead in most regions. It appeared that in general, 
counties focused on their own schools, although in two RESAs, (2 and 5), there was a more 
regional approach to developing and using EEA-trained trainers as local experts available 
across counties. Although two-thirds of the participants in the EEAs were teachers, they 
seemed to face more obstacles in both delivering and receiving additional training due to the 
real limits of county budgets for supporting release time. Most RESAs provided additional 
training to EEA-trained trainers in their regions, but a minority of them as of May 2014 has 
firm plans for providing additional support going forward. Most RESA PD directors appreci- 
ated the technical expertise of WVDE trainers that was exhibited at the EEAs and hope to 
continue to draw upon that expertise. While as a group they considered the EEAs an effective 


1 /Hi I Educator Enhancement Academies Evaluation Study 


Executive Summary 


way to roll out the standards some expressed concern about sustaining the effort over the time 
it will take to fully implement the NxGen CSOs. 

EQ4. What was the NxGen professional development experience of teachers during the 
2013-2014 school year? 

Whether they were EEA-trained or end-user teachers, relatively few teachers received 
professional development of a duration (at least 30 contact hours) that one would expect to 
produce changes in teacher practice or student performance. Further, it was relatively rare for 
end-user teachers to meet for NxGen training more than three times between June 2013 and 
May 2014. EEA-trained teachers fared better in that regard, with about half meeting more 
than three times. 

For the most part, end-user teachers were trained by district or school staff (or both). 
EEA-trained teachers were quite likely to also receive training from RESA staff. 

Most teachers reported that their training did not include time to plan classroom im- 
plementation of NxGen CSOs or to practice new skills. Teachers generally felt they had not 
met frequently enough, nor did they have adequate contact hours to learn the skills and con- 
tent. They did engage in their training collectively, which research shows increases the chances 
of teachers gaining common understandings and collaborating back in their schools. They 
could also see the alignment of the training they received with their own schools' and districts' 
goals. 

EQ5. What were the outcomes of 2013-2014 professional development experiences in 
terms of teachers' overall sense of preparedness to teach the NxGen CSOs and perceived 
impacts of these experiences on knowledge, practice, and beliefs? 

At the end of the school year preceding full implementation of the NxGen standards, 
just over a third of end-user teachers viewed themselves as well prepared to teach to the 
NxGen standards. Among EEA-trained teacher trainers, nearly half shared that level of confi- 
dence. Notably, more than a quarter of all teachers responding to the survey considered them- 
selves not at all prepared. 

In addition to reporting their sense of preparedness, teachers also provided assess- 
ments— both before (retrospectively) and after their training during 2013-2014— of the exten- 
siveness of their knowledge of the standards, practice of the instructional shifts, and belief 
that their students can achieve at levels benchmarked in the standards. Overall, they believed 
themselves to have gained the most from their training in their knowledge of the standards, 
less in their practice of the instructional shifts, and least of all in their beliefs about the poten- 
tial success of their students in reaching the NxGen benchmarks. On average, they began and 
ended the year believing “to a small extent” that their students could reach the benchmarks. 
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EQ6. What training-related factors may have been at work to produce these outcomes? 

When working in combination, district, school, and RESA staff were more likely than 
when working alone, to provide higher quality, more frequent, and longer duration profes- 
sional development-all qualities associated in this study with heightened confidence in being 
prepared to teach the NxGen Standards. 

Recommendations 

While a close study of the findings in this study may point to additional needs, we make 
three major recommendations: 

Strongly focus on raising trainers' and teachers' expectations and beliefs that their students 
can learn at levels benchmarked in the Next Generations standards. 

The professional development that was offered during 2013-2014 did little to convince 
teachers that their students could learn at higher levels. Yet decades of research has shown 
the impact teacher expectations can have on their students achievement and the benefits of 
academic press, so this is a critical area of need. 

Focus future train-the-trainer activities on district level staff 

The success of the train-the-trainer model depends on the ability of those trained to 
provide training back in their home school or district. In this study we learned that teachers 
and principals were at a disadvantage to provide such training, especially in terms of schedul- 
ing the number of hours and follow-up meetings that research indicates it takes to change 
teacher practice and improve student performance. District central office staff were in a much 
better position to provide such training and, in fact, did provide most of the training received 
by teachers in the targeted grades across the state. 

Provide an infrastructure for ongoing training and access to guidance materials for local 
trainers 

After providing their initial training back in their home districts, many participants in 
the EEAs reported needing more training themselves and help in locating guidance resources 
such as sample lesson plans, pacing guides, and the like. Very few of the EEA-trained teachers 
had received professional development of sufficient duration to effect changes in their own 
practice, and only about half considered themselves fairly well or very well prepared to teach 
to the standards themselves, let alone train others to teach to the standards. 
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Introduction 

As the final stage of West Virginia’s rollout of the Next Generation Content Standards 
and Objectives (NxGen CSOs), 1 the regional education service agencies (RESAs) conducted six 
train-the-trainer events to prepare educators— mostly teachers— to provide professional de- 
velopment back in their home schools and districts. These events, called Educator Enhance- 
ment Academies (EEAs), lasted two or three days depending on which RESA conducted them, 
and targeted teachers in grade levels that had not yet received professional development in 
the NxGen CSOs, that is, Grades 2-3, 6-8, and 10-12. 2 

The first phase of this study looked at how well prepared those trainers were at the end 
of their EEA experience by asking them a series of questions in two surveys about their expe- 
riences during the training and after they, themselves, conducted training sessions during the 
summer of 2013. The first survey, administered at the conclusion of the EEA sessions included 
a knowledge test to gauge trainers’ understanding of the instructional shifts involved in teach- 
ing to the new standards. 

We found that the EEAs provided important components of a coherent instructional 
system by focusing on the new NxGen standards and instructional shifts needed to teach to 
the standards, and by introducing participants to materials and tools at their disposal in their 
own trainings and classrooms. The design of the EEAs reflected three of five elements of re- 
search-based professional development, including having a strong content and content peda- 
gogy focus; alignment with school, district, and state goals; and active learning. 

The knowledge test, however, revealed a need for additional professional development 
for these frontline trainers, as overall they answered correctly only two thirds of the items. The 
lowest scores were for middle school mathematics (58% correct) and the highest were for el- 
ementary mathematics (78% correct). For additional details about knowledge test results, see 
our Phase 1 report (Hammer & Hixson, 2014, pages 17-19). 

Findings also showed differences between the four EEAs led by content experts from 
the West Virginia Department of Education (WVDE) compared with the two led by Corwin 
Press. For example, participants at the WVDE-led trainings were much more likely than those 
at Corwin trainings to indicate that the training had been a good start and they were looking 
forward to training others or that the training had provided everything they needed to train— 
by a margin of about 2.5 to 1. Conversely, attendees at Corwin-led events were about 12 times 
more likely to indicate they did not feel ready to train others. In our discussion of these find- 
ings we suggested that expectations for the training may not have been clearly laid out for the 
Corwin trainers, and that contracts or memoranda of understanding for future training in- 
clude specific expectations and standards to be met. 


1 The NxGen CSOs are West Virginia’s adaptation of the Common Core State Standards. 

2 Two of the EEAs were each sponsored by two RESAs operating in partnership; consequently 
there were six EEAs, which served eight regions. 
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Methods 


While our first investigation looked at the quality of the experience of participants in 
the EEAs and how well those events prepared them to train others, this report further exam- 
ines the experience of those same participants in providing their own training, what additional 
professional development they received from the RESAs, and the experience of the end-user 
teachers who received professional development from the EEA-trained teachers and other 
sources in the targeted grade levels during the 2013-2014 school year. 

Research Questions 

EQi. To what extent did participants in the EEAs follow up with their own training? 

EQ2. What challenges did EEA participants encounter and what supports did they indicate 
they needed going forward? 

EQ3. What follow-up did RESAs provide after the initial EEA training? 

EQ4. What was the NxGen PD experience of end-user teachers and EEA teacher trainers 
during the course of the 2013-2014 school year? 

EQ5. What were the outcomes of 2013-2014 professional development experiences in terms 
of teachers’ overall sense of preparedness to teach to the NxGen CSOs and perceived 
impacts of those experiences on knowledge, practice, and beliefs? 

EQ6. What training-related factors may have been at work to produce these outcomes? 

Methods 


Population Characteristics and Sampling Procedures 

We examined three populations in this phase of the Educator Enhancement Acade- 
mies (EEA) study: (a) 953 teachers, district office staff, and others who received training at 
the RESA-sponsored EEAs to become teacher trainers ; (b) professional development directors 
or executive directors from the eight RESAs; and (c) general and special education teachers 
involved in teaching English/language arts (ELA) and mathematics across the state in Grades 
2-3, 6-8, and 10-12. For all groups, we surveyed the full population. 

Data Collection Methods 

Using three instruments, we collected data as follows: 

• For EEA trainees, we used the Follow-up EEA Participant Survey in September 2013; 

• For RESA professional development directors, the RESA PD Director Interview Pro- 
tocol (May 2014); and 

• For ELA and math teachers in targeted grades across the state, we used the NxGen 
Standards Professional Learning Survey (April-May 2014). 
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Research Design 

Our data collection and methods of analysis are summarized by research question in 
Table l below. 

Table 1. Summary of Evaluation Questions and Data Collection Methods Used 


Method of analysis/doto 
Evaluation question source 

Results reported 

EQ1. To what extent did Descriptive statistics/ 
participants in the EEAs Follow-up EEA Participant 
follow up with their own Survey (September 2013) 
training? 

• How many attendees provided training? 

• For which programmatic levels and contents areas 
did they train individuals? 

• How many individuals attended? 

• How many training events did they facilitate? 

• How many additional hours did they expect to 
provide for typical participant? 

EQ2. What challenges Qualitative data analysis/ 
did EEA participants Follow-up EEA Participant 

encounter and what Survey (September 2013) 

supports did they 
indicate they needed 
going forward? 

For those EEA attendees who did provide training, 

• What challenges have EEA attendees encountered 
in their own trainings? 

• What specific areas/topics from the NxGen CSOs 
were most difficult for participants to understand? 

• What was the duration of the training for typical 
attendee? 

• About which three topics (if any) could trainers have 
used more information or assistance in their own 
trainings 

• In what other ways can RESAs be of additional 
assistance? 

For those EEA attendees who did not provide training, 

• What were the reasons? 

• Were there other supports that would have made it 
possible for them to provide training? 

EQ3. What follow-up did Qualitative data analysis/ 
RESAs provide after the RESA PD Director Interview 
initial EEA training? Protocol (May 2014) 

• In what contexts did participants deliver their own 
trainings in their regions? School-based? County- 
based? Centralized RESA-based? 

• Have EEA participants asked for additional 
assistance from RESAs to support their own 
training? 

• What follow-up did RESAs do with the trainers 
trained at the EEAs they hosted? 

• Did RESAs expect to do any additional follow-up 
with the trainers trained at their EEAs? If so, what 
form will that take? 

• What additional supports do the RESAs need from 
the West Virginia Department of Education? 

• Were there comments about the EEAs or the roll- 
out and implementation of the NxGen CSOs? 

• What, if any, evaluation have RESAs conducted of 
the implementation or effectiveness of NxGen CSO 
professional development in their regions? 


Table 1 continued next page 
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Table 1. Summary of Evaluation Questions and Data Collection Methods Used, continued 


Method of analysis/doto 

Evaluation question source Results reported 


EQ4. What was the 
NxGen PD experience of 
end-user teachers and 
EEA teacher trainers 
during the course of the 
2013-2014 school year? 

they received on NxGen standards? 

• To what extent did teachers receive research-based 
training? 

• How well prepared do teachers consider 
themselves to teach to the NxGen CSOs? 


Descriptive statistics/A/xGen • What was the average number of PD contact hours 


Standards Professional 
Learning Survey (April-May 
2014) 


teachers in targeted grades received on NxGen 
standards? 

• What was the average number of times teachers in 
targeted grades met for PD on NxGen standards? 

• Who was primarily responsible for providing the PD 


EQ5. What were the 
outcomes of training in 
terms of overall sense 
of preparedness and 
impacts on knowledge, 
practice, and beliefs? 
EQ6. What training- 
related factors may 
have been at work to 
produce these 
outcomes? 


Paired samples t tests and 
Cohen's d/NxGen Standards 
Professional Learning Survey 
(April-May 2014) 


• What was the perceived impact of the NxGen 
training teachers received on their knowledge, 
behavior, and beliefs? 

• Did it differ for the two groups (i.e., EEA teacher 
trainers and end-user teachers? 


Pearson's r and probability 
testing/NxGen Standards 
Professional Learning Survey 
(April-May 2014) 


• Is there a relationship between total hours of PD 
contact hours received and how well prepared 
teachers believe themselves to be? 

• Is there a relationship between the number of times 
teachers met for training and how well prepared 
they believe themselves to be? 


• Is there a relationship between the level of 
adherence to research-based PD practices and how 
well prepared teachers believe themselves to be to 
teach to the NxGen standards? 


Comparison of means/ 
NxGen Standards 
Professional Learning Survey 
(April-May 2014) 


• Are certain trainer role groups associated with 
greater teacher confidence about their 
preparedness to teach to the NxGen standards? 

• How are trainer role groups associated with 
variations in hours of training time provided, 
number of times met, and adherence to research- 
based PD practices? 


Both surveys were administered using similar strategies. First a message was sent via 
e-mail to the full list of intended respondents, announcing the survey and explaining its pur- 
poses, importance, and the time it takes to fill out; and asking recipients to watch for it and 
respond quickly. SurveyMonkey was then used to e-mail the invitation to participants, along 
with an embedded link to the online questionnaire. The initial invitation message and three 
reminder messages were uploaded to SurveyMonkey and sent to nonrespondents every 4 or 5 
business days. In all, it was possible for individuals to receive up to five messages, including 
the announcement, the initial invitation, and three reminders if they failed to respond. This 
technique yielded large enough respondent pools to produce statistically significant and reli- 
able results in most cases, even when respondents were disaggregated to represent different 
subgroups (e.g., middle school math or elementary school ELA teachers). To determine the 
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number of teachers we needed in our statewide survey, we used the MaCorr Research Solu- 
tions Online sample size calculator. 

Results 

Tables supporting the various results in this section can be found in the Appendix. 
Following the Reference section results are presented by evaluation question. 

Recruitment 

Of the 953 EEA participants, 599 responded to the Follow-up EEA Participant Survey, 
for a 63% response rate. Similarly, of the 4,686 ELA and math teachers in the targeted grades, 
participants returned 1,662 usable responses to the NxGen Standards Professional Learning 
Survey, for a 25% response rate. Although this is a lower response rate than we typically see- 
probably due to testing and other pressures on teachers during the April-May survey period- 
the total number of responses fell only slightly short of our calculated target sample size 
(1,740). PD directors or executive directors responded to the interview protocol for all eight 
RESAs. 


Statistics and Data Analysis 

EQl. To what extent did participants in the EEAs follow up with their own training? 

The great majority of the respondents (85%) to the September 2013 follow-up survey 
indicated they had provided training. Of the nearly 500 respondents who had provided train- 
ing, each trained an average of 56 educators. District central office staff reported training the 
largest numbers of teachers, averaging 165 each; general classroom teachers trained the small- 
est numbers of educations, averaging about 32 each. The total number of teachers these re- 
spondents trained collectively is unknown since it is possible co-trainers reported training the 
same groups of teachers, thus double counting attendees. 

The respondents each held an average of 2.6 training events and the duration of the 
training for a typical participant was about 9 hours. District central office staff tended to offer 
more hours of training for the individuals they trained (i.e. 12 hours), while instructional sup- 
port teachers/ specialists (non-special education) provided the briefest training sessions (i.e., 
7.6 hours). General classroom teachers reported providing 9.2 hours of training on average. 

About two thirds of those who had provided training planned to provide additional 
follow-up training or already had; they anticipated that this additional training would last an 
average of another 10 hours or so. About 87% of district central office staff intended to provide 
an average of another 17 hours, which when combined with their initial offerings (12 hours) 
would total about 29 hours— a duration very close to the 30 hours research has shown to be 
the minimum needed to change teacher practice and/or improve student performance (Yoon 
et al., 2007). On the other hand, less than two thirds of general classroom teachers— who made 
up the largest segment of the EEA trainees— planned to provide follow-up training. Of those 
who did, they anticipated offering an average of about 8 additional hours. This time along with 
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the initial training of about 9 hours adds up to just over half of the recommended 30 contact 
hours. 

For more detailed information on the various role groups trained in the EEA, and the 
average numbers of (a) participants in their trainings, (b) training events they have held, (c) 
hours of training they have provided, (d) and additional hours they plan to provide, see Table 
A 1, page 23. 

EQ2. What challenges did EEA participants encounter and what supports did they indicate 
they needed going forward? 

The Follow-up EEA Participant Survey (September 2013) included several open-ended 
questions requesting information about trainers’ experiences and needs going forward. The 
qualitative responses were read three times and coded. This section summarizes what was 
learned from responses to the open-ended questions. 

Eighty-seven respondents indicated they had not provided training as of mid-to-late 
September 2013. When asked to describe the reasons for not providing training, the most fre- 
quent response was that training had not been requested (n = 20) or they encountered various 
scheduling issues (n = 18), including scheduling conflicts, a lack of time in the schedule for it, 
or that it was scheduled for later in the year. Some (n = 11) indicated that others besides them- 
selves had provided training or that most of the teachers from their school had attended the 
EEA (n = 7); the remainder had various personal reasons for not providing training (change 
of position, health problems, and so forth). 

EEA attendees who had succeeded in providing training were also asked about barriers 
they encountered. Some indicated they had encountered no barriers (n = 50). The most fre- 
quent response, however, indicated scheduling time with teachers was the biggest barrier (n 
= 119); while others mentioned they were having trouble getting people’s attention (n = 43); 
they lacked standards-aligned curriculum materials, lessons, and strategies (n = 34); or they 
had an overall need for more knowledge about how to teach the NxGen CSOs (n = 32). 

When the EEA attendees who had provided training were asked about what additional 
supports they needed from RESAs to help them with their own training, 182 respondents in- 
dicated they needed no additional help from the RESAs, while 215 indicated that RESAs could 
continue to provide help. From the latter, we received 273 suggestions. The most frequently 
mentioned (n = 105) was follow-up training, especially on the standards and instructional 
shifts, which some (n = 21) said they needed on an ongoing basis throughout the year. Two 
other frequently mentioned needs were for additional resources (n = 55) such as NxGen 
aligned curriculum and instruction resources or model lessons, and time for collaboration and 
planning (n = 34). 

When asked about what topics (if any) were the most difficult for individuals in their 
sessions to master, the most frequently mentioned topics (in descending order) were the ap- 
plication of instructional shifts (n = 86); how to teach to NxGen standards effectively, includ- 
ing lesson planning; differences among NxGen CSOs, previous CSOs, and the Common Core 
(n = 36); identifying and obtaining aligned curriculum materials and texts (n = 29); Smarter 
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Balanced assessment (n = 29); and how to help students transition from the old CSOs to 
NxGen while addressing their knowledge and skills gaps (n = 22). 

EQ3. What follow-up did RESAs provide after the initial EEA training? 

RESA directors responded to an interview protocol in early May 2014, which was de- 
signed to collect information about RESA support of the trainers trained at the EEAs, subse- 
quent to the academies and throughout the 2013-2014 school year. We asked about the 
context of training that the EEA-trained trainers provided, what sorts of supports the trainers 
may have requested from the RESAs, any follow-up RESAs provided or planned to provide 
specifically for the EEA-trained trainers in their regions, as well as what additional supports 
might be needed from the WVDE. We also gave directors an opportunity to comment generally 
about the rollout and implementation of the NxGen CSOs in their region and to describe the 
nature of any evaluations they were conducting. 

Training contexts 

RESA directors mostly described activities that took place at the county level (de- 
scribed below). However, staff from two RESAs did mention school-based training led by 
EEA-trained trainers at the beginning of the school year and at school retreats focused on 
particular NxGen topics. Another director mentioned school-based training that took place on 
early-release days and in-service education (ISE) and continuing education (CE) days. 

At the county level, RESA PD directors reported a variety of approaches: 

• County-based centralized summer training academies utilizing EEA-trained trainers; 

• Ongoing, county-organized, grade-level meetings that featured training on NxGen 
CSOs; 

• Partnerships among counties to hold training events that were at least partially led by 
the EEA-trained trainers; 

• Regional cooperation and cross-county sharing of EEA-trained trainers-usually cen- 
tral office staff-who have developed strong expertise in particular NxGen topics; and 

• Ongoing county-supported work with school-based PLCs, which meet for 1 or 2 hours 
in the morning 1 day a week, when the students have a delayed start. 

One RESA conducted three Educator Enhancement Academies for educators across the re- 
gion, utilizing EEA-trained trainers; they also provided a leadership series to prepare princi- 
pals to lead the implementation of NxGen CSOs in their schools. 

Trainer requests for support and additional training 

When asked about the nature of training and support requested by EEA-trained train- 
ers in their regions, the picture is less clear. RESA staff mostly responded with descriptions of 
the training that counties had requested, perhaps because such requests were filtered through 
the county offices. Another possibility is that EEA-trained trainers who were teachers could 
not get release time during school. One director reported that superintendents were strongly 
resistant to providing release time due to the added expense and the need to keep talented 
teachers in the classroom with students. Providing ongoing support for such school-based 
trainers may be a special challenge that needs looking into. 
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Training and support provided to trainers 

RESA PD directors were also asked what follow-up training they had provided to the 
EEA-trained trainers. Some had provided several additional days of training as illustrated in 
the following examples: 

• two days for EEA-trained trainers to work with their central office staffs on implemen- 
tation and training plans; 

• monthly trainings and formal one-day sessions targeted to both ELA and math train- 
ers; 

• support for attendance at a 4-day quality learning workshop and three full-day ses- 
sions for EEA-targeted content/programmatic teams in the fall of 2013 and winter of 
2014; 

• funding (with WVDE grant money) to contract with 18 of the original 33 EEA-trained 
trainers to sustain region-wide training related to the implementation of the NxGen 
CSOs in classrooms; 

• a series of three leadership trainings for administrators who attended the EEA; and 

• quarterly 6-hour trainings for trainers to deepen their understanding of the CSOs and 
instructional practices that support the next generation CSOs. 

To address the challenge of supporting trainers unable to get release time, RESA 2 provided 
extensive online libraries of NxGen resources and ran webinars. This same RESA supported a 
portal and centralized regional training (four sessions) for EEA-trained math coaches and is 
supporting teams working on implementation monitoring tools. 

One final note, a RESA pointed out that, although they have provided additional train- 
ing for both ELA and math trainers, there is no guarantee that counties will continue to send 
the same trainers. 

Planned additional follow-up for EEA-trained trainers 

Some of the RESAs had definite plans to continue working with and supporting the 
EEA-trained trainers in their regions. Perhaps the most spelled out plan was RESA 5, which 
is contracting with at least 16 EEA-trained teachers and teacher coaches from all eight coun- 
ties in their region to continue to provide support for implementation of the NxGen CSOs in 
2014-2015. Other RESAs mentioned additional training for segments of their EEA-trained 
trainers, such as instructional coaches, or for groups that may include both previously trained 
trainers and newly appointed ones. Three RESAs had no definite plans for ongoing support of 
EEA-trained trainers, indicating only that they will respond to county requests. 

Additional support needed from WVDE 

Seven of the eight RESAs indicated their desire for ongoing collaboration with and 
support from WVDE, although one director mentioned the need for familiarity with new ex- 
pertise available at the Department, due to personnel changes. Three directors mentioned the 
need for funding to support release time for teacher trainers. Others mentioned the need for 
technical expertise, such as the Department provided at the EEAs, to support additional train- 
ing in writing lesson plans, locating curriculum resources, specific ELA or math instructional 
approaches, and other NxGen implementation issues in classrooms and schools. 
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Overall view on the EE As as a mechanism for rolling out and implementing the NxGen CSOs 

The RESAs that used WVDE staff to provide training at their academies generally ex- 
pressed great satisfaction with the EEAs and considered them very successful (the two RESAs 
that used third-party trainers did not provide comments about the EEAs or rollout). Most of 
the RESAs urged the Department to continue to work with the RESAs on implementing the 
standards, which many directors noted was going to take some time and sustained effort. Sev- 
eral of the RESAs described their efforts to further develop local or regional experts by, for 
example, supporting PLCs in schools or by sustaining a team of trainers who can cross county 
lines and provide specific training. 

Regional evaluations of the implementation or effectiveness of this approach to PD 

None of the RESAs seem to be specifically studying the implementation or effective- 
ness of the EEA train-the-trainer approach to NxGen professional development. They all col- 
lect feedback from participants using their standardized evaluation form after trainings. RESA 
2 described a more complex approach to data collection in their region, including support for 
learning team dialogues about what was and was not working. They and RESA 6 are develop- 
ing tools to monitor implementation of the standards for principals, teachers, and trainers. 
RESA 2 also hired a third-party evaluator to do research on their trainings, including pre- 
/posttests. 

EQ4. What was the NxGen professional development experience of teachers during the 
2013-2014 school year? 

All ELA and math teachers in the targeted grades (i.e., Grades 2-3, 6-8, and 10-12) for 
whom we had email addresses (4,686) were invited to participate in the NxGen Standards 
Professional Learning Survey during April-May 2014. The survey asked an array of questions 
about teachers’ experiences in professional development since the preceding June. The re- 
maining analyses in this report are based on the approximately 1,600 responses we received 
to the survey. Unless otherwise specified, the findings are reported for two groups of teachers - 
those who had attended an EEA to become teacher trainers and other teachers who partici- 
pated in professional development facilited by EEA-trained teachers and others (hereafter 
end-user teachers. 

Number of PD contact hours teachers received 

The survey asked how much training teachers received from June 2013 through April- 
May 2014, a time period subsequent to the EEAs, which took place in April and May 2013. 
Teachers who had attended the EEAs to become trainers received substantially more hours of 
training afterwards (an average of 23 hours) compared with end-user teachers in the targeted 
grades, who received about 12 hours. Figure 1 provides a comparison of duration categories 
for both groups of teachers. EEA-trained teachers were about three times more likely to meet 
at the research-based recommended level of 30 or more hours. For a breakdown by program- 
matic level, content area, and region, see Table A 2 (page 24) and Table A 3 (page 25). 


Educator Enhancement Academies Evaluation Study / 9 


Results 


Percent of responses by hours of training 
June 2013 through May 2014 


26 


35 


27 

EEA-trained teachers 
(n = 303) 


11 


19 

0 hours 

40 

1-4 hours 


5-13 hours 

19 

10 

^ 1 

14-29 hours 

■ 30+ hours 

End-user teachers 


(n = 1,080) Percent of respondents 


Percent of responses by number of times met June 
2013 through May 2014 


Number of times teachers met 
for PD 

Similar to findings for 
the number of hours of train- 
ing, the two groups of teachers 
varied in the number of times 
they met. EEA-trained teach- 
ers met an average of 5 times, 
compared with 3.6 times for 
end-user teachers. Figure 1 
shows the comparison by 
meeting frequency category. 

It is notable that more than 
half of end-user teachers re- 
ported meeting two or fewer 
times to learn about NxGen 
CSOs. For a breakdown by 
programmatic level, content 
area, and region, see Figure 1 
(page 24) and Table A 3 (page 
25)- 

Role groups primarily responsi- 
ble for providing the PD 

Teachers were asked 
to select up to three sources of 
NxGen training from among 
six options (i.e., “Our princi- 
pal or assistant/associate 
principal,” “A teacher from 
my school,” “RESA staff,” 

“District staff,” “Vendor,” or 
“Other”). Of all the possible 
combinations, the seven 
shown in Figure 2 were the 
most prevalent, representing 
about 80% of all responses. As 
a reminder, the RESAs were 

charged with facilitating the EEAs in the spring of 2013 and most had provided extensive fol- 
low-up with the trainers who were trained at those events. Consequently, EEA-trained teach- 
ers received more training from RESAs, either as their sole source or in combination with 
school or district staff during subsequent months and into the 2013-2014 school year than 
did end-user teachers. The great majority of end-user teachers received their NxGen training 
exclusively from district or school staff, or from a combination of the two. 
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Figure 1. Duration and Frequency of Professional Development for 
EEA-Trained and End-User Teachers 

EEA-trained teachers received more hours of training and met more 
frequently than end-user teachers during the same time period, June 
2013 through May 2014. They were three times more likely to meet 
for 30 or more hours, and about 1.5 times more likely to meet four or 
more times than end-user teachers. 
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Extent to which teachers received re- 
search-based training 

Teachers were asked to re- 
spond to 15 items, the Research- 
Based PD Practices Scale, which to- 
gether provided a measure of the ex- 
tent to which the professional 
development they received had ad- 
hered to five qualities research has 
found to be associated with improve- 
ments in teacher practice and/or stu- 
dent performance (Hammer, 2013). 
The five qualities— content and con- 
tent-pedagogy focus, active learning, 
coherence, collective participation, 
and sufficient duration and 
timespan— were represented by three 
items each. Overall, the mean per- 
centage of agreement (agreed/ 
strongly agreed) with the items was 
66.0% for EEA-trained teachers ver- 
sus 55.7% for end-user teachers. 
Among teachers of different content 
areas, special education teachers 
tended to give higher marks for these 
items than teachers of ELA or math 
and middle school teachers gave 
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Figure 2. Sources of NxGen Training by Teacher Category 
This chart shows the most prevalent combinations of one 
or more sources of NxGen training reported by respondents 
to the NxGen Standards Professional Learning Survey in 
April-May 2014, representing about 80% of respondents. 
The remaining 20% cited other combinations of sources 
with much lower frequencies. The teacher-only category is 
broken out separately here, but is actually a subset of the 
within-school only category. 


higher marks than elementary and 

high school teachers. EEA-trained teachers from RESAs 3 and 6 averaged well above the over- 
all mean among EEA-trained teachers, while end-user teachers in RESAs 2, 3, 5, and 6 all gave 
above the mean ratings for adherence to research-based PD practices in the NxGen training 
they had received (see Table A 5, page 27 for additional details). 
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Figure 3 shows 
the breakdown by indi- 
vidual item in the 
Research-Based PD 
Practices Scale. For 
both groups of teachers 
(EEA-trained and end- 
user), there was general 
agreement that the 
training was aligned 
with school and district 
goals, that it involved 
collective participation 
with their colleagues at 
school or in the district, 
and that the training 
was about the correct 
length of time for the 
content it aimed to 
cover. Some areas 
where both groups were 
similar in their disa- 
greement included that 
the training allowed 
time to plan classroom 
implementation, that it 
included practice of new 
skills, and that it in- 
cluded an adequate 
number of meetings 
over time. All three of 
these items pertain to 
actual implementation 
of the NxGen standards, 
thus the finding that the 
training teachers re- 
ceived did not allow 
time for these activities could affect teachers’ sense of how prepared they are to teach to the 
standards (see next section). Overall, the EEA-trained teachers were in stronger agreement 
than the end-user teachers that various elements of research-based PD practice were present 
in the training they received. 
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Figure 3. Adherence to Research-Based Practices by Individual Item, EEA- 
Trained and End-User Teachers 

Ns for EEA-trained teachers ranged from 278 to 288, depending upon the items; 
for end-user teachers the range was from 918 to 940 responses. Only teachers 
who indicated they had participated in at least 1 hour of training since June 2013 
were included in the analysis. 
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EQ5. What were the outcomes of 2013-2014 professional development experiences in 
terms of teachers' overall sense of preparedness to teach the NxGen CSOs and perceived 
impacts of these experiences on knowledge, practice, and beliefs? 

Level of preparedness 

Teachers were asked to rate their level of preparedness to teach to the NxGen CSOs in 
the 2014-2015 school year on a Likert -type scale with o = not at all, 1 = somewhat, 2 = mod- 
erately, 3 = fairly well, and 4 = very well. Looking first at the teachers one would expect to be 
the best prepared— those trained to be NxGen trainers at the EEAs— Figure 4 shows a surpris- 
ing 31% of the ELA teacher trainers and 29% of math teacher trainers believed themselves not 
at all prepared to teach to the NxGen CSOs. On the other hand, nearly half of this group of 
teacher trainers consid- 
ered themselves fairly or 
very well prepared, com- 
pared with just over a 
third of end-user teach- 
ers who shared that same 
degree of confidence. 

Looking at mean 
scores on the Likert scale 
described above, teach- 
ers trained to be trainers 
at the EEAs scored at the 
moderately prepared 
level (2.0) for both ELA 
and math. End-user 
teachers scored a bit be- 
low that level with mean 
scores of 1.8 for both 
ELA and math. Details of 
these preparedness rat- 
ings can be found in Ta- 
ble A 6 (page 28). 

Impacts on knowledge, practice, and beliefs 

The questionnaire included three sets of questions, which asked teachers to rate their 
levels of (a) knowledge about the NxGen CSOs, (b) practice of the instructional shifts required 
to effectively teach to the standards, and (c) belief that their students can perform at levels 
benchmarked in the standards— before and after the training they had received during the 
study period. Results for both EEA-trained and end-user teachers, displayed in Figure 5, were 
statistically significant and revealed large effects on knowledge about the NxGen standards, 
moderate effects on respondents’ practice of the instructional shifts called for in the NxGen 
standards, and small effects on teacher beliefs that their students can perform at levels bench- 
marked in the NxGen standards (see Table A 7, page 29). Teacher trainers reported moving 
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Figure 4. Level of Preparedness to Teach to the NxGen CSOs for Teacher 
Trainers and End-user teachers of ELA and math. 

EEA-trained teacher trainers reported very similar levels of preparedness 
whether they taught ELA or math. The same was true of end-user teachers. This 
analysis included only teachers who indicated they had received at least 1 hour 
of NxGen training since June 2013. 
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Figure 5. Pre-Post Retrospective Perception of Impact on 
Knowledge, Practice, and Belief 

The ratings in this chart include 0 (not at all); 1 (to a small 
extent); 2 (to a moderate extent); and 3 (to a great extent). 
While teacher trainers' pretest ratings (in aqua) were similar for 
all three measures (knowledge, practice, and belief), they 
reported greater gains in their knowledge and practice of 
NxGen standards than for their belief in their students' ability 
to achieve at levels benchmarked in the standards. The same 
pattern held true for end-user teachers, although they started 
at lower levels and gained less. Especially notable is the failure 
to rise above "to a small extent" in end-user teachers' beliefs 
about their students' potential performance. 

their students can perform at the levels benchmarked in 
dally true of math and special education teachers. 


from fairly low-levels of knowledge 
of the standards, practice of the in- 
structional shifts, and beliefs in 
their students' abilities to achieve 
the standards before the training 
they received during the 2013-2014 
school year, to moderate levels after 
the training. End-user teachers, on 
the other hand, started from com- 
paratively lower levels and moved 
up only slightly to just below mod- 
erate levels for their knowledge and 
practice. Their beliefs about their 
students' ability to perform at levels 
benchmarked in the NxGen stand- 
ards remained at a low level, that is, 
they believed "to a small extent" 
that their students could hit those 
benchmarks. 

We wondered how much 
the results of this measure varied by 
content area, so we conducted the 
same analysis for all teachers who 
reported at least 1 hour of training 
(Figure 6). Differences in the pre- 
post scores were statistically signif- 
icant at the .001 level (see Table A 
8, page 30). What may be most no- 
table about this chart is the fairly 
low-level of belief teachers held that 
the NxGen standards. This was espe- 


When these data were subjected to the Cohen’s d effect size test, the differences in the 
impact of training on the three teacher outcomes became clearer. As noted in the horizontal 
axis of Figure 7, effects can range from small (less than .4) to moderate (.5 to .7) to large (.8 
and above). Overall, the training teachers received had, by their reckoning, a larger effect on 
their knowledge about the standards than on the other two measures, with the smallest effects 
on teachers’ beliefs about their students’ ability to perform to levels benchmarked in the 
NxGen standards. Teachers of both ELA and math (mostly elementary teachers) reported the 
largest effects on all three measures, followed by special education teachers on two of the 
measures (knowledge and practice). Math and special education teachers seemed to be least 
impacted with regard to raising their expectations about their students’ ability to perform to 
the standards. 
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Figure 6. Pre-Post Retrospective Perception of Impact on Knowledge, Practice, and Belief by Content 
Area 

The ratings in this chart include 0 (not at all); 1 (to a small extent); 2 (to a moderate extent); and 3 (to a 
great extent). 



All teachers who received at least 1 hour of training were included in this test of effect size. Changes 
between retrospective pre- and posttest reports of knowledge, practice, and beliefs were statistically 
significant. Effect sizes varied greatly, from small for changes in beliefs to quite large for changes in 
knowledge. 
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EQ6. What training-related factors may have been at work to produce these outcomes? 


At this point in our reporting we have described the range of teachers’ experiences with 
regard to the training they received, and the range of reported outcomes in terms of their sense 
of preparedness. Next we explore relationships between the two sets of observations. 


Duration of training and 
number of times met 

The relation- 
ship between the num- 
ber of hours of training 
and preparedness was 
relatively weak and 
statistically nonsignifi- 
cant for ELA (r = .044; 
p = .094); for math, 
however, the correla- 
tion was a bit stronger 
and statistically signif- 
icant (r = .204; p = 
.000; see Table A 9, 
page 31). Figure 8 
graphically displays 
these relationships. 
The most common du- 
ration category was 
training lasting a half 
day to about two days 
(5-13 hours). Results 
for this duration of 
training fell just below 
the moderately pre- 
pared level for both 
math and ELA. Ac- 
cording to these data 
ELA teachers did not 
gain confidence from 
additional hours of 
training, whereas math 
teachers did, especially 
if they had 30 or more 
contact hours. 

Shifting to the 
number of times met, 
the relationship is 
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Figure 8. Preparedness to Teach to the NxGen Standards by Duration and 
Frequency of Training 

The top chart shows the mean preparedness rating assigned by ELA or math 
teachers who reported receiving varying amounts of training (from 0 to more 
than 30 hours). The bottom chart shows the mean preparedness rating assigned 
by teachers who met various numbers of times for training. 
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clearer as shown in Figure 8. As the number of times met rose, so did the mean level of pre- 
paredness in nearly all cases. These results were statistically significant for both ELA (r = .147; 
p = .000) and math (r = .174; p = .000). 


Use of research-based 

Figure 9 
shows that teachers 
who attended train- 
ing that included 
70% or more of 
practices associated 
with effective pro- 
fessional develop- 
ment tended to 
consider themselves 
about twice as pre- 
pared as those 
whose training that 
included 20% or 
fewer of those prac- 
tices. This positive 
correlation was 
found to be statisti- 
cally significant and 
stron-ger than ei- 
ther the duration or 
frequency of train- 
ing for both ELA (r 
= .308; p = .000) 
and math (r = .292; 
p = .000; see Table 
A 9, page 31). 


PD practices 


Research-based practices 
ELA teachers ■ Math teachers 
0% - 10% (ELA n = 50; math n = 47) 

11% - 20% (ELA n = 132; math n = 130) 

21% - 30% (ELA n = 93; math n = 88) 

31% - 40% (ELA n = 129; math n = 119) 

41% - 50% (ELA n = 87; math n = 88) 

51% - 60% (ELA n = 155; math n = 151) 

61% - 70% (ELA n = 82; math n = 81) 

71% - 80% (ELA n = 154; math n = 151) 

81% - 90% (ELA n = 148; math n = 144) 

91% - 100% (ELA n = 213; math n = 207) 

0 12 3 4 

Level of preparedness (0 = not at all; 2 = moderately; 4 = very well 



Figure 9. Preparedness to Teach to the NxGen Standards by Use of Research-Based 
PD Practices 

Teachers responded to 15 items in the survey meant to measure the adherence to 
PD practices found to be associated with changes in teacher practice and student 
achievement. This graph shows that when teachers reported the PD they attended 
had higher percentages of the research-based practices, they also tended to report 
higher levels of confidence in their preparedness to teach to the NxGen standards. 


Role group(s) of trainers 

Specific role groups or combinations of role groups were associated with mean prepar- 
edness levels as shown in Figure 10. Overall, receiving training from some combination of 
RESA staff, district staff, or one’s own school staff seemed to result in higher levels of confi- 
dence that teachers’ were prepared to teach to the NxGen standards than receiving training 
from only one of these role groups. Figure 10 breaks out the fellow teacher subset from the 
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larger own school staff 
category because teachers 
were such a large segment 
of the EEA participants. 
Yet end-user teachers 
who received their train- 
ing only from a fellow 
teacher were at the bot- 
tom of the ranks for re- 
porting a sense of prepar- 
edness. Receiving train- 
ing solely from their own 
school staff or from a 
RESA staff member alone 
ranked only slightly high- 
er than training solely 
from a fellow teacher. 

A chi-square test 
showed that the differ- 
ences among role groups 
were statistically signifi- 
cant for all but the follow- 
ing combinations: “own 
school staff and RESA 


l , Math preparedness rating 

District and RESA staff (n = 30 ELA, 
31 math) 

Own school staff and district staff 
(n = 145 ELA, 144 math) 

Own school staff and RESA staff (n 
= 23 ELA; 24 math) 

District staff (n = 307 ELA, 296, 
math) 


RESA staff (n ■ 


Own school staff (n = 
math) 

Fellow teacher (n ■ 
math) 


55 ELA, 52 math) 
203 ELA, 199 


99 ELA, 99 


l ELA preparedness rating 



0.0 1.0 2.0 3.0 

Preparedness rating 


4.0 


Figure 10. Preparedness to Teach to the NxGen Standards by Role Group(s) 
of Trainers 

This analysis includes all end-user teachers who received at least 1 hour of 
training. The fellow teacher category is a subset of own school staff , broken 
out here because such a large segment of the trainers trained in the EEAs 
were teachers, making up about two thirds of all EEA trainees. 


staff’, “district staff and RESA staff”, and “own school staff and district staff’ (math only; see 
Table A 10, page 31). 


Relationship of role group(s) of trainers to duration and frequency of training, and use of research- 
based practices. 

To better understand the variation in preparedness outcomes among trainer role 
groups, we looked at the differences for role groups in their use of research -based PD prac- 
tices, the number of times they met with end-user teachers, and the duration of training they 
provided. Table 2 shows the relatively lower rates at which the fellow teacher only, school 
staff only, and district staff only role groups provided training adhering to research -based PD 
practices, lasting more than about two days, or that met at least four times. Yet, these three 
role groups acting alone provided training to most of the respondents in the survey. 
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Table 2. Differences Among Training Sources Regarding Use of Research-Based Practices, Duration of 
Training, and Number of Times Met 


Percent of research-based PD 

practices used 

Duration in hours 

Number of times met 

English/language arts 


Training received from . . . 
District and RESA staff 
School and district staff 
School and RESA staff 
RESA staff only 
District staff only 
School staff only 
Fellow teacher only 

Training received from . . . 
District and RESA staff 
School and district staff 
School and RESA staff 
RESA staff only 
District staff only 
School staff only 
Fellow teacher only 



SD 

27.76 

29.78 

24.48 

30.46 

28.60 

31.31 

29.36 


N 

79 

211 

60 

137 

413 

290 

149 


Mean 


69.4 



SD 

26.63 

28.58 

26.71 

27.93 

28.80 

30.75 

29.34 


N Mean 



Math 

N Mean 



SD 

15.58 

14.66 
13.82 
12.45 
14.94 
10.73 

10.66 

SD 

16.93 

13.84 

14.54 

17.17 
16.12 
11.19 

12.17 



SD 

3.11 

5.26 

7.69 

3.43 

4.30 
4.56 
2.93 

SD 

3.47 

5.22 

6.64 

3.54 

4.31 
5.03 
3.11 


Discussion 

Findings are summarized and interpreted below by evaluation question. 

EQ1. To what extent did participants in the EEAs follow up with their own training? 

The great majority of the respondents (85%) to the September 2013 follow-up survey 
indicated they had provided training. Depending upon EEA participants’ regular role in their 
counties, they reported providing more or less training to end-user teachers. District central 
office staff seemed in the best position to provide training as evidenced by the numbers they 
trained, the number of hours they had provided to a typical participant in their sessions by 
early fall 2013, and the additional hours of training they planned to provide. By comparison, 
other role groups appeared at a disadvantage, especially teachers and principals with regard 
to planned follow-up. 

EQ2. What challenges did EEA participants encounter and what supports did they indicate 
they needed going forward? 

Most EEA-trained trainers were able to schedule at least some training; where training 
had taken place, scheduling was still difficult, as was getting people’s attention and buy-in. 
Many of the trainers reported needing more training themselves, especially in the instruc- 
tional shifts and other aspects of classroom implementation, and help in locating curriculum 
and other resources they needed for planning instruction. 
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EQ3. What follow-up did RESAs provide after the initial EEA training? 

EEA-facilitated follow-up training took place in schools, counties, and RESAs, alt- 
hough counties seemed to be squarely in the lead in most regions. It appeared that in general, 
counties focused on their own schools, although in two RESAs (2 and 5), there was a more 
regional approach to developing and using EEA-trained trainers as local experts available 
across counties. Although two-thirds of the participants in the EEAs were teachers, they 
seemed to face more obstacles in both delivering and receiving additional training due to the 
real limits of county budgets for supporting release time. Most RESAs provided additional 
training to EEA-trained trainers in their regions, but a minority of them as of May 2014 has 
firm plans for providing additional support going forward. Most RESA PD directors appreci- 
ated the technical expertise of WVDE trainers that was exhibited at the EEAs and hope to 
continue to draw upon that expertise. While as a group they considered the EEAs an effective 
way to roll out the standards some expressed concern about sustaining the effort over the time 
it will take to fully implement the NxGen CSOs. 

EQ4. What was the NxGen professional development experience of teachers during the 
2013-2014 school year? 

Whether they were EEA-trained teachers or end-user teachers, relatively few teachers 
received professional development of a duration (at least 30 contact hours) that one would 
expect to produce changes in teacher practice or student performance. Further, it was rela- 
tively rare for end-user teachers to meet for NxGen training more than three times between 
June 2013 and May 2014. EEA trained teachers fared better in that regard, with about half 
meeting more than three times. 

For the most part, end-user teachers were trained by district or school staff (or both). 
EEA-trained teachers were quite likely to also receive training from RESA staff. 

Most teachers received training that did not include time to plan classroom implemen- 
tation of NxGen CSOs or to practice new skills. Teachers generally felt they had not met 
enough times, nor did they have enough contact hours to learn the skills and content. They 
did engage in their training collectively, which research shows increases the chances of teach- 
ers gaining common understandings and collaborating back in their schools. They could also 
see the alignment of the training they received with their own schools' and districts' goals. 

EQ5. What were the outcomes of 2013-2014 professional development experiences in 
terms of teachers' overall sense of preparedness to teach the NxGen CSOs and perceived 
impacts of these experiences on knowledge, practice, and beliefs? 

At the end of the school year preceding full implementation of the NxGen standards, 
just over a third of end-user teachers viewed themselves as well prepared to teach to the 
NxGen standards. Among EEA-trained teacher trainers, nearly half shared that level of confi- 
dence. Notably, more than a quarter of all teachers responding to the survey considered them- 
selves not at all prepared. 
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In addition to reporting their sense of preparedness, teachers also provided assess- 
ments— both before (retrospectively) and after their training during 2013-2014— of the exten- 
siveness of their knowledge of the standards, practice of the instructional shifts, and belief 
that their students can achieve at levels benchmarked in the standards. Overall, they believed 
themselves to have gained the most from their training in their knowledge of the standards, 
less in their practice of the instructional shifts, and least of all in their beliefs about the poten- 
tial success of their students in reaching the NxGen benchmarks. On average, they began and 
ended the year believing “to a small extent” that their students could reach the benchmarks. 

EQ6. What training-related factors may have been at work to produce these outcomes? 

When working in combination, district, school, and RESA staff were more likely than 
when working alone, to provide higher quality, more frequent, and longer duration profes- 
sional development-all qualities associated in this study with heightened confidence in being 
prepared to teach the NxGen Standards. 

Recommendations 

While a close study of the findings in this study may point to additional needs, we make 
three major recommendations: 

Strongly focus on raising trainers' and teachers' expectations and beliefs that their students 
can learn at levels benchmarked in the Next Generations standards. 

The professional development that was offered during 2013-2014 did little to convince 
teachers that their students could learn at higher levels. Yet decades of research has shown 
the impact teacher expectations can have on their students achievement and the benefits of 
academic press, so this is a critical area of need. 

Focus future train-the-trainer activities on district level staff 

The success of the train-the-trainer model depends on the ability of those trained to 
provide training back in their home school or district. In this study we learned that teachers 
and principals were at a disadvantage to provide such training, especially in terms of schedul- 
ing the number of hours and follow-up meetings that research indicates it takes to change 
teacher practice and improve student performance. District central office staff were in a much 
better position to provide such training and, in fact, did provide most of the training received 
by teachers in the targeted grades across the state. 

Provide an infrastructure for ongoing training and access to guidance materials for local 
trainers 

After providing their initial training back in their home districts, many participants in 
the EEAs reported needing more training themselves and help in locating guidance resources 
such as sample lesson plans, pacing guides, and the like. Very few of the EEA-trained teachers 
had received professional development of sufficient duration to effect changes in their own 
practice, and only about half considered themselves fairly well or very well prepared to teach 
to the standards themselves, let alone train others to teach to the standards. 
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Table A 1. Training Provided and Planned by EEA Attendees Reported by Role Group 


Will provide follow-up 

Provided training training 







Mean 



% of 

Additional 






number 

Typical 


those 

hours 


Total 



Mean 

training 

duration 


that had 

training/ 


respon- 



number 

events 

of 


provided 

support 

Role groups 

dents 

n 

% 

trained 

held 

training 

n 

training 

planned 

All role groups 

578 

493 

85.3 

55.8 

2.57 

9.3 

338 

68.6 

10.3 

District central office 

staff 

63 

56 

88.9 

165.3 

4.54 

12.0 

49 

87.5 

17.2 

General classroom 

teacher 

378 

318 

84.1 

31.9 

2.27 

9.2 

202 

63.5 

8.0 

Instructional Coach 

18 

16 

88.9 

81.3 

2.80 

9.0 

14 

87.5 

8.4 

Instructional support 
teacher/specialist 
(non-special educa- 
tion) 

63 

58 

92.1 

70.3 

2.96 

7.6 

47 

81.0 

11.9 

Principal/assistant 

principal 

27 

24 

88.9 

62.1 

1.65 

8.9 

15 

62.5 

11.1 

Special education 

29 

21 

72.4 

41.3 

1.90 

8.9 

11 

52.4 

6.5 


teacher 

Data source: Follow-up EEA Participant Survey (September 2013) 

Means include only those EEA trainers who indicated they had provided training and who planned to provide 
follow-up. 
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Table A 2. End-User Teachers: Mean Hours of PD Received and Number of Times Met June 2013 Through 
May 2014 

Hours of PD received Number of times met 




Mean 

n 

SD 

Mean 

n 

SD 

Total 


12.1 

1080 

14.214 

3.6 

1079 

6.076 


Targeted grade levels 





Elementary School (Grades 2-3) 


10.8 

382 

11.150 

3.7 

381 

4.904 

Middle School (Grades 6-8) 


13.5 

409 

17.545 

3.8 

409 

7.530 

High School (Grades 10-12) 


11.0 

339 

11.878 

3.3 

338 

4.833 



Content areas 





English/language arts 


11.4 

256 

12.795 

3.3 

257 

4.605 

Mathematics 


15.2 

243 

17.203 

4.0 

244 

9.158 

Both English/language arts and mathematics 

11.7 

206 

12.004 

3.9 

204 

5.679 

Special education 


11.0 

282 

15.136 

3.5 

281 

4.507 

Other 


9.7 

93 

8.909 

3.6 

93 

4.241 



RESA 






RESA1 


10.6 

113 

12.757 

2.9 

112 

3.089 

RESA2 


14.4 

94 

12.790 

4.1 

94 

4.268 

RESA3 


12.4 

151 

14.252 

4.4 

151 

10.272 

RESA4 


13.5 

71 

16.773 

4.9 

72 

9.715 

RESA5 


15.6 

110 

15.463 

3.3 

110 

3.873 

RESA6 


12.4 

75 

13.337 

4.6 

76 

5.395 

RESA7 


12.1 

211 

17.715 

3.8 

212 

5.886 

RESA8 


9.7 

182 

10.345 

2.5 

181 

3.144 


NOTE: Pink highlights lower than mean numbers for the subgrouping, while green indicates mean or higher 
than mean numbers. 
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Table A 3. EEA-Trained Trainers: Mean Hours of PD Received and Number of Times Met June 2013 
Through May 2014 

Hours of PD received Number of times met 


Teacher respondents in . . . 



Mean 

n 

SD 

Mean 

n 

SD 

Total 



23.1 

303 

20.758 

5.0 

309 

5.750 


Targeted grade levels 





Elementary School (Grades 2-3) 



18.3 

65 

15.679 

4.3 

67 

4.426 

Middle School (Grades 6-8) 



22.9 

123 

22.541 

5.2 

125 

5.523 

High School (Grades 10-12) 



25.1 

127 

20.545 

5.1 

129 

6.387 



Content areas 






English/language arts 



20.3 

109 

17.823 

4.9 

110 

6.030 

Mathematics 



27.9 

116 

24.234 

5.4 

119 

5.146 

Both English/language arts and mathematics 


19.3 

36 

16.233 

4.6 

37 

4.113 

Special education 



21.6 

29 

21.153 

5.2 

30 

8.924 

Other 



17.9 

13 

13.203 

3.7 

13 

3.301 




RESA 






RESA1 



24.5 

31 

19.218 

4.7 

33 

6.728 

RESA2 



26.6 

21 

21.630 

5.6 

21 

4.631 

RESA3 



22.8 

43 

26.837 

6.8 

44 

6.391 

RESA4 



23.6 

20 

19.999 

4.4 

20 

2.889 

RESA5 



24.1 

17 

18.079 

4.8 

17 

4.918 

RESA6 



28.4 

34 

24.229 

5.4 

34 

3.620 

RESA7 



11.8 

56 

14.623 

2.7 

57 

2.781 

RESA8 



32.8 

53 

18.120 

5.7 

55 

6.884 


NOTE: Pink highlights lower than mean numbers for the subgrouping, while green indicates higher than 
mean numbers. 


Educator Enhancement Academies Evaluation Study / 25 




Appendix 


Tables Displaying Results 


Table A 4. Adherence to Research-Based Practices by Individual Item, EEA-Trained and End-User Teachers 


EEA-trained teachers 


End-user teachers 


PD research-based practices 


Respon- 

dents 

this Agreed or strongly agreed 
item Number 


Percent 


Respon- 

dents 

this Agreed or strongly agreed 
item Number 


Percent 


c 

CD 

■4-J 

c 

o 

u 


o 

u 


Content focused 285 

Content-pedagogy focused 284 

Used curriculum materials in 285 

training 

Aligned with school and district 286 

goals 

Aligned to individual learning 285 

needs 

Was challenging and not 278 

redundant with previous training 



937 

933 

932 

926 

933 

918 



Included discussions 288 

Included time to plan classroom 286 

implementation 

Included practice of new skills 287 

and feedback 



931 


931 


932 


493 


300 


386 


53 


32 


41 


o 

u 


Involved collective participation 284 

with colleagues from own school 
Facilitate collaboration with 286 

colleagues in work setting 
Developed common 284 

understandings among 



936 


932 


940 


788 


540 


525 



o 


Had sufficient contact hours to 285 

learn new skill and content 

Included adequate number of 285 

meetings over time to practice 

Was about the correct length of 279 

time for the content 



929 


920 


921 


407 


367 


654 



NOTE: This analysis includes only respondents who indicated they had received at least 1 hour of NxGen training 
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Table A 5. Adherence to Research-Based Practices Mean Score by Content Area, Programmatic Level, and 
Home RESA: EEA-Trained Versus End-User Teachers 



EEA-trained teachers 

End-user teachers 


Mean n SD 

Mean n SD 

Overall 

66.0 288 26.522 

55.7 951 29.687 

Content area 


ELA 

Math 

Both ELA and Mathematics 
Special education 
Other 

Elementary school (Grades 2-3) 
Middle school (Grades 6-8) 

High school (Grades 2-3) 

RESA1 

RESA2 

RESA3 

RESA4 

RESA5 

RESA6 

RESA7 

RESA8 



55.1 

225 

29.056 

53.1 

219 

29.072 

51.7 

190 

29.789 

61.1 

238 

29.464 

57.5 

79 

31.783 


55.0 

343 

30.449 

58.2 

349 

28.873 

53.8 

296 

30.030 


52.0 

96 

30.641 

63.8 

86 

29.222 

58.9 

138 

29.945 

55.2 

61 

31.914 

57.5 

94 

30.237 

63.8 

66 

27.244 

52.8 

186 

28.960 

50.9 

160 

27.892 


NOTE: Includes only respondents who indicated they had received at least 1 hour of NxGen training. 
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Table A 6. Level of Preparedness to Teach to the NxGen CSOs for EEA-Trained and End-User Teachers 


Level of preparedness 

Teachers trained to train at EEA 

End-user teachers 


Mean 

rating 

Number 

Percent 

Mean 

rating 

Number 

Percent 

Prepared to teach ELA 

All respondents 

2.03 

275 

100.0 

1.81 

982 

100.0 

0 = Not at all prepared 


85 

30.9 


253 

25.8 

1 = Somewhat 


24 

8.7 


166 

16.9 

2 = Moderately 


38 

13.8 


209 

21.3 

3 = Fairly well 


54 

19.6 


224 

22.8 

4 = Very well 


74 

26.9 


130 

13.2 

Prepared to teach math 

All respondents 

2.00 

273 

100.0 

1.76 

962 

100.0 

0 = Not at all prepared 


79 

28.9 


263 

27.3 

1 = Somewhat 


24 

8.8 


167 

17.4 

2 = Moderately 


43 

15.8 


192 

20.0 

3 = Fairly well 


71 

26.0 


215 

22.3 

4 = Very well 


56 

20.5 


125 

13.0 
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Table A 7. Pre-Post Retrospective Measure of Impact, Including Means, Correlation, and Effect Size by 
Group 

Comparison group Treatment group Paired- 



(Pretest rating) 


(Posttest rating) 


Pre/ 

samples t 

Effect 

Description 

Mean 

SD 

N 

Mean 

SD 

N 

Posttest 

correlation 

test sig. 
(2-tailed) 

size 

d 

Teachers trained to train at EEA 

Knowledge about NxGen 
standards 

1.5 

0.857 

286 

2.3 

0.641 

286 

.518 

.000 

1.035 

Practice of the instruc- 
tional shifts 

1.6 

0.920 

282 

2.1 

0.716 

282 

.590 

.000 

0.623 

Belief students can per- 
form at benchmarked 

levels 

1.4 

0.792 

283 

1.7 

0.733 

283 

.682 

.000 

0.377 

End-user teachers 

Knowledge about NxGen 
standards 

1.3 

0.778 

946 

1.9 

0.658 

946 

.520 

.000 

0.836 

Practice of the instruc- 
tional shifts 

1.3 

0.875 

917 

1.8 

0.784 

917 

.616 

.000 

0.504 

Belief students can per- 
form at benchmarked 

levels 

1.2 

0.793 

922 

1.4 

0.814 

922 

.768 

.000 

0.212 


Note: The rating scale for the three pre-post retrospective impact items included 0 ( not at all), 1 (to a small 
extent), 2 (to a moderate extent), and 3 (to a great extent). 
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Table A 8. Significance Testing and Effect Size Analysis of Impacts on Knowledge, Practice, and Beliefs by 
Content Area 

Paired Differences 

95% Confidence 
Std. Std. interval of the 
devia- error difference 
Mean tion mean Lower Upper 


English/language arts teachers (n = 356) 


Knowledge about 
NxGen standards 

.63483 

.72874 

.03862 

.55887 

.71079 

16.437 

355 

.000 

0.7263 

Practice of the in- 
structional shifts 

.44318 

.75648 

.04032 

.36388 

.52248 

10.991 

351 

.000 

0.4260 

Belief students can 
perform at bench- 
marked levels 

.28490 

.57943 

.03093 

.22407 

.34573 

9.212 

350 

.000 

0.2737 



Mathematics teachers 

(n = 351) 





Knowledge about 
NxGen standards 

.63818 

.71524 

.03818 

.56309 

.71326 

16.716 

350 

.000 

0.8197 

Practice of the in- 
structional shifts 

.44643 

.71034 

.03875 

.37020 

.52266 

11.520 

335 

.000 

0.4374 

Belief students can 

.25588 

.51736 

.02806 

.20069 

.31107 

9.120 

339 

.000 

0.2116 


perform at bench- 
marked levels 


Both English/language arts and mathematics teachers (teach both subjects, n = 249) 


Knowledge about 

.70281 

.71281 

.04517 

.61384 

.79178 

15.558 

248 

.000 

1.0895 

NxGen standards 

Practice of the in- 

.62500 

.76387 

.04851 

.52946 

.72054 

12.885 

247 

.000 

0.7866 

structional shifts 

Belief students can 

.29960 

.59752 

.03802 

.22471 

.37448 

7.880 

246 

.000 

0.3071 


perform at bench- 
marked levels 


Special education teachers (n = 299) 


Knowledge about 

.65552 

.72704 

.04205 

.57277 

.73826 

15.591 

298 

.000 

1.0691 

NxGen standards 

Practice of the in- 

.47603 

.70548 

.04129 

.39477 

.55728 

11.530 

291 

.000 

0.5288 

structional shifts 

Belief students can 

.23469 

.55670 

.03247 

.17079 

.29859 

7.229 

293 

.000 

0.2092 


perform at bench- 
marked levels 

This analysis includes all teachers who reported receiving at least 1 hour of training. 


Sig. 

( 2 - 

t df tailed) Cohen's d 
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Appendix 


Tables Displaying Results 


Table A 9. Relationships of Duration (Hours), Times Met for Training, and Use of Research-Based PD 
Practices to Teachers' Preparedness to Teach ELA or Math (Pearson's r) 




Prepared to 
teach ELA 

Prepared to 
teach math 

Duration in hours 

Pearson Correlation 

.044 

.204 


Sig. (2-tailed) 

.094 

.000 * 


N 

1473 

1433 

Number of times met for training 

Pearson Correlation 

.147 

.174 


Sig. (2-tailed) 

.000 * 

.000 * 


N 

1476 

1438 

Research-based practices standardized score 

Pearson Correlation 

.308 

.292 


Sig. (2-tailed) 

.000 * 

.000 * 


N 

1311 

1271 

Note: This analysis includes all teachers. 
“"Statistically significant at the .001 level 





Table A 10. Chi-Square Test of Categorical Variable "Role Group of Trainer(s)" and Preparedness to Teach 
ELA and Math 



Prepared to teach ELA 


Prepared to teach math 

X 2 

df 

P 


X 2 

df 

P 

Fellow teacher 

23.911 

4 

.000 

* 

21.392 

4 

.000 * 

Own school staff 

15.550 

4 

.004 

* 

29.932 

4 

.000 “* 

District staff 

19.947 

4 

.001 

* 

10.174 

4 

.038 * 

RESA staff 

13.469 

4 

.009 

* 

13.166 

4 

.010 * 

District staff and RESA staff 

5.536 

4 

.237 


6.903 

4 

.141 

Own school staff and RESA staff 

2.212 

4 

.697 


1.838 

4 

.765 

Own school staff and district staff 

13.115 

4 

.011 

* 

7.494 

4 

.112 


Note: This analysis includes only teachers who received at least 1 hour of training. 
“"Statistically significant at the .05 level 
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Appendix 


Tables Displaying Results 


Table A 11. Preparedness to Teach to NxGen Standards in ELA or Math by Extensiveness of training 


Prepared to teach ELA 

Prepared to teach math 



Mean 



Mean 


Duration categories 

N 

rating 

SD 

N 

rating 

SD 

No training (0 hours) 

98 

1.22 

1.281 

105 

1.26 

1.264 

Informational (1-4 hours) 

179 

1.26 

1.250 

188 

1.58 

1.223 

Technical training (5-13 hours) 

389 

1.86 

1.345 

388 

1.99 

1.374 

Sustained PD (14-29 hours) 

181 

1.91 

1.501 

195 

1.98 

1.427 

Recommended (30 or more hours) 

105 

2.46 

1.316 

98 

1.81 

1.571 

Note: The preparedness rating scale used for this item was 0 = 

not at all; 1 

= somewhat, 2 

= moderately, 3 = 

fairly well, and 4 = very well. 







Table A 12. Preparedness to Teach to NxGen Standards in ELA or Math by Number of Times Met for 


Training 








Prepared to teach ELA 

Prepared to teach math 



Mean 



Mean 


Number of times met for training 

N 

rating 

SD 

N 

rating 

SD 

None 

177 

1.35 

1.253 

168 

1.23 

1.242 

1 time 

137 

1.41 

1.309 

141 

1.40 

1.282 

2 times 

186 

1.83 

1.269 

ITT 

1.62 

1.361 

3 times 

155 

1.76 

1.433 

151 

1.80 

1.410 

4-5 times 

148 

2.19 

1.337 

148 

2.09 

1.367 

6 or more 

170 

2.31 

1.452 

167 

2.39 

1.383 


Table A 13. 

Preparedness to Teach to NxGen Standards in ELA or Math by Percent of Research-Based 
Practices Used in Training 

Percent of re 
practices 


Prepared to teach ELA 

Prepared to teach math 

search-based PD 

N 

Mean 

rating 

SD 

N 

Mean 

rating 

SD 

0% - 10% 


50 

1.14 

1.229 

AT 

1.15 

1.179 

11% - 20% 


132 

1.17 

1.162 

130 

1.18 

1.103 

21% - 30% 


93 

1.51 

1.239 

88 

1.50 

1.269 

31% - 40% 


129 

1.60 

1.349 

119 

1.61 

1.296 

41% - 50% 


87 

1.77 

1.387 

88 

1.67 

1.337 

51% - 60% 


155 

2.01 

1.410 

151 

1.82 

1.410 

61% - 70% 


82 

2.18 

1.508 

81 

1.99 

1.401 

71% - 80% 


154 

2.17 

1.423 

151 

2.04 

1.505 

81% - 90% 


148 

2.36 

1.366 

144 

2.47 

1.343 

91% - 100% 


213 

2.41 

1.492 

207 

2.30 

1.500 
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